Evaluation of speech synthesis systems for Dutch in telecommunication applications

An evaluation was conducted to compare four TTS systems for Dutch with respect to intelligibility and acceptability, as a follow-up to an evaluation study conducted in 1997. Intelligibility was measured by the ability of listeners to write down correctly semantically unpredicatable sentences. Acceptability was tested in two ways: by asking listeners for subjective judgments on a number of semantic scales for individual TTS systems, and by asking for preference judgments in a pairwise comparison. For the acceptability tests a weather forecast and an e-mail text were used as test materials. It was found that the system giving best intelligibility performed worst in the acceptability tests. It was concluded that subjective acceptability is not a simple consequence of intelligibility, and that a distinction needs to be made between the esthetic and functional dimensions of synthetic speech. In comparing the results of the current study with those of the evaluation study conducted in 1997, it appeared fair to conclude that at least two systems are available for Dutch which constitute a substantial improvement over last year’s state of the art, although it is evident from the results that many aspects are still in need of improvement.